Skip to content

[YouTube] PoToken implementation with HtmlUnit#12027

Closed
Stypox wants to merge 3 commits intoTeamNewPipe:devfrom
Stypox:htmlunit-po-token
Closed

[YouTube] PoToken implementation with HtmlUnit#12027
Stypox wants to merge 3 commits intoTeamNewPipe:devfrom
Stypox:htmlunit-po-token

Conversation

@Stypox
Copy link
Copy Markdown
Member

@Stypox Stypox commented Feb 10, 2025 •
edited
Loading

What is it?

  • Bugfix (user facing)
  • Feature (user facing)
  • Codebase improvement (dev facing)
  • Meta improvement to the project (dev facing)

Description of the changes in your PR

This is an experimental PR that aims to provide an alternative PoTokenProvider implementation that does not rely on the system WebView. HtmlUnit is apparently able to simulate a browser well enough, and the generated poTokens work, however there are a couple of problems:

  • The code does not compile unless minSdk is increased to 26 due to this issue. The APK built by this PR has minSdk increased to 26 for testing purposes, but this is obviously not the solution.
  • minifyEnabled seems to create issues for release builds.
  • While HtmlUnit seems to work without issues in an emulator, on my phone the debug version uses a lot of CPU and after some time the system terminates the app. The release version works fast enough though. Let me know if it works on your phone, try both the debug (from CI) and the release APK (provided below).
  • HtmlUnit uses a forked version of Gecko which might conflict with the extractor one.

APK testing

The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR. You can find more info and a video demonstration on this wiki page.

Release APK built at commit 768db9f with minifyEnabled=false (otherwise it wouldn't build); it might be faster than the debug version: app-release.zip

Due diligence

@github-actions github-actions Bot added the size/large PRs with less than 750 changed lines label Feb 10, 2025
@Isira-Seneviratne
Copy link
Copy Markdown
Member

Isira-Seneviratne commented Feb 10, 2025

It seems the solution to the build issue is to disable Instant Run.

@gechoto
Copy link
Copy Markdown

gechoto commented Feb 12, 2025

What is the benefit of using HtmlUnit compared to the WebView?

@Stypox
Copy link
Copy Markdown
Member Author

Stypox commented Feb 13, 2025

HtmlUnit would be a platform-independent solution that does not rely on the system webview. It could be an alternative for those phones where the WebView implementation is broken or results in invalid poTokens. However, this is mostly an experimental PR, since there are a few problems highlighted above with no clear solution, plus the HtmlUnit library would increase the APK size.

@Isira-Seneviratne
Copy link
Copy Markdown
Member

Isira-Seneviratne commented Feb 13, 2025 •
edited by Stypox
Loading

The system webview is updatable starting with Android 5.0, so this PR might not be necessary.

@Stypox
Copy link
Copy Markdown
Member Author

Stypox commented Feb 13, 2025 •
edited
Loading

That is true in theory, but we all know device manufacturers tend to mess with default Android components more than they should... But yeah this might not be worth it.

@ale5000-git
Copy link
Copy Markdown

ale5000-git commented Feb 17, 2025 •
edited
Loading

The system webview is updatable starting with Android 5.0, so this PR might not be necessary.

The WebView by Google is now Android 8.0+, so other devices are stuck with old WebViews and websites may break on them sooner or later.
There are also other WebViews but the common user would be unable to get them.

@Isira-Seneviratne
Copy link
Copy Markdown
Member

Isira-Seneviratne commented Feb 17, 2025

The system webview is updatable starting with Android 5.0, so this PR might not be necessary.

The WebView by Google is now Android 8.0+, so other devices are stuck with old WebViews and websites may break on them sooner or later. There are also other WebViews but the common user would be unable to get them.

In that case, it might be a good idea to bump the app's minimum SDK version to 26.

@ale5000-git
Copy link
Copy Markdown

ale5000-git commented Feb 17, 2025 •
edited
Loading

In that case, it might be a good idea to bump the app's minimum SDK version to 26.

Personally I'm strongly against it, specially for this app.
People may keep old phones for music (and YouTube).

@ale5000-git
Copy link
Copy Markdown

ale5000-git commented Feb 20, 2025

@Stypox
Isn't possible to have the "Gecko" library as separate app (doesn't need to be a system WebView, so it will be easy to install it)?
In this way NewPipe (or even other apps if programmed to do it) can use it if it is installed, and it won't increase NewPipe size for the users that doesn't use it.

@Stypox
Copy link
Copy Markdown
Member Author

Stypox commented Feb 20, 2025

@ale5000-git yes that would be a good idea

@ShareASmile ShareASmile added feature request Issue is related to a feature in the app discussion This needs to be discussed before anything is done code quality Improvements to the codebase to improve the code quality labels Apr 8, 2025
@Stypox
Copy link
Copy Markdown
Member Author

Stypox commented Sep 30, 2025

Closing for now, this was more of a (kind-of-successful) proof-of-concept

@Stypox Stypox closed this Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

code quality Improvements to the codebase to improve the code quality discussion This needs to be discussed before anything is done feature request Issue is related to a feature in the app size/large PRs with less than 750 changed lines

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Stypox @Isira-Seneviratne @gechoto @ale5000-git @ShareASmile